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ABSTRACT 



This study is part of the Flashlight Project, a Web-based 
system for creating surveys, gathering responses, and analyzing results. The 
purpose of this study was to assess Web- supported classes (fully and partly 
online) in 10 community colleges in California. Faculty members coordinated 
the study on their respective campuses; a total of 710 students were 
surveyed. Results indicated that students who are newly exposed to electronic 
communication are more likely to have greater enthusiasm and active 
participation in classes than if they are in face-to-face classes. Students 
also reported that e-mail and online discussion enhanced their relationship 
with the instructors. Students are more likely to articulate suggestions and 
complaints to the instructors, and they are more likely to receive detailed 
comments from the instructors in online classes. However, students report 
that they are less likely to discuss academic goals and career plans with 
their instructors in online classes than in face-to-face classes. When 
comparing student responses based on whether they had experienced electronic 
communication in a fully online or partly online course, the study found that 
classes that combine both face-to-face and electronic communication may be 
more useful and exciting to more students than the fully online classes. (GC) 
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Abstract 



The project’s purpose was for students and faculty to assess web-supported classes (fully and 
partly online) in the ten participating community colleges. Preliminary indications are that 
students usually new to classes using electronic communication, say they are more likely to 
actively participate in classes with these tools than in face-to-face classes. They also indicate a 
greater willingness to ask clarifying questions when they don’t understand something, and they 
are will more willing to discuss ideas and concepts with others students in the class — ^when using 
electronic communication. These students say that email and online discussion has enhanced 
their relationship to the instructor. They are more likely to receive detailed comments from the 
instructor more promptly than in a face-to-face course. And they are more likely to tell the 
instructor that they have a complaint or suggestion than in a face-to-face course. Electronic 
communication, however, has its limits: students indicated they were less likely to discuss 

academic goals and career plans with their instructor nor were they likely to discuss ideas and 
concepts taught in the course with their instructor. The net result was that classes relying fully or 
partly on electronic communication generated a great deal of student enthusiasm. Comparing 
student responses based on whether they had enrolled in fully online or partly online classes 
suggests that the combination of face-to-face and electronic communication may be more useful 
and exciting for more students than the fully online classes. 
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Purpose 



The project’s purpose was for students and faculty to assess web-supported classes (fully and 
partly online) in the ten participating community colleges. 

College Teams 

In each college, participating faculty (usually 3 per college) received a stipend of $ 1 ,000 for 
attending two workshops, giving students the cross-college survey, conducting their own course- 
specific assessment, and submitting a final report. One researcher per college was contributed by 
the colleges. The selection of the faculty came mostly from the college Chief Instructional 
Officers. The grant provided for three faculty per college, but some colleges paid stipends for 
additional faculty where interest was high. See faculty and researcher agreement forms. 
Appendix G. 

Workshops 

The national Flashlight Project's director, Steve Ehrmann, with Alec Testa, Western Governor’s 
University (WGU), conducted one-day workshops, one north and one south, in October for 
planning the assessment activities; Dr. Ehrmann’s colleague, Robin Zuniga conducted one-day 
north and south workshops in May for reviewing survey and project results. Faculty found the 
spring workshop more useful than the fall workshop. This may be due to the fact that in the 
spring, faculty looked at concrete survey responses for their own students, their college, and 
across the colleges (workshop agendas. Appendix G). 

Faculty Online Discussion 

We had planned for topical conference discussions on the project web site, moderated by Alec 
Testa, WGU. As it turned out, the faculty preferred a listserv for discussion and announcements. 
The web site seemed less convenient than finding the news and comments in email. Dan 
Mitchell at DeAnza provided the listserv. 

The Flashll2ht Student Survey 

At the fall workshops, and during the ensuing online conversation, the participants agreed to 
narrow the focus of the survey to students’ uses and perceptions of electronic communication. 
We included email, online conferences, chat rooms, and news groups — among students or 
between faculty and students. We contracted to use items fi:om the Current Student Inventory in 
the extensive Flashlight Handbook, developed and tested by the national group in Washington. 
The survey was conducted in mid April for semester courses and in late May for quarter system 
courses. We surveyed 710 students. 

We beta tested the Flashlight-Silhouette program from Washington State University; the program 
allows students to submit responses directly fi:om their class web site. Gary Brown and his 
excellent staff returned aggregate data (within and across colleges) and individual class data to 
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each project participant. Rio Hondo’s Mike Martinez and Peg Collins at WSU solved program 
glitches quickly and helped faculty remedy problems on their web site. 

Key Survey Results 

(Datasets from the survey are available by request. Contact Dr. Robert Gabriner: rgabrine@ccsf.org) 

SECTION I: Findings Across All Students 

1. Among the 710 students who responded to the survey, over half indicated that the use of 
electronic communication helped them participate in the course; 

• 50% said they actively participate in scheduled discussions about course material, 

• 60% said they asked for clarification when they did not understand something, 

• 50% said they discussed ideas and concepts taught in the class with other students. 

2. Students said that using electronic communication enhanced their relationship to their 
instructor; 

• 58% said they were more like to receive detailed comments on assignments from their 
instructor in the course using electronic communication rather than a face-to-face course, 

• 56% said they were more likely to receive comments from the instructor more quickly 
than in a face-to-face course, 

• 50% said they were more likely to tell the instructor that they had a complaint or a 
suggestion about the course than in a face-to-face course. 

3. Students also indicated they were less likely to discuss their academic goals and career plans 
with their instructor in fully online or partially online classes; nor were they as likely to 
discuss the ideas and concepts taught in the course with the instructor. 

4. Students were more likely to be more autonomous in classes using electronic communication 
than in face-to-face classes; 

• 56% said that they were more likely to search for answers to their questions rather than 
ask the instructor or other students in classes using electronic communication. 

5. Classes using electronic communication appeared to generate a significant level of student 
enthusiasm; 

• 8 1 % said that they put more thought in their comments, 

• 72% said they were more comfortable asking awkward questions, 

• less than 30% said they had difficulty relating to other students in the class, 

• 61% said they spent more time studying, 

• 74% said they were more confident that they would be able to reach their academic goals. 

6. Students expressed a great deal of satisfaction with the way technology was used in their 
classes; that is, 

• 90% said they found the technology appropriate for performing tasks in the course, 

• Over 90% said they would recommend both the course and the instructor to other 
students, 

• 91% said they would recommend that other students take classes with electronic 
communications, 

• 84% expressed overall satisfaction with the classes they took using electronic 
communications . 
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7. Students participating in this survey rated themselves high in ability to use electronic 
technology; that is, 

• 78% said they could send and receive email, 

• 70% said they could search for information on the Internet, 

• 56% said they knew how to send and receive electronic files by way of a computer. 

SECTION II: Comparing Students in Fully Online Classes to Partly Online Classes 

1. When comparing student responses based on whether they had experienced electronic 
communication in a fully online or partly online or web-enhanced course, we found some 
significant differences: 

• Students in partly web-enhanced or partly online classes were more likely to ask for 
clarification from the instructor when they didn’t understand something. 

• They were also more likely to work on assignments with other students than those who 
were fully online. 

• Students in fully online classes, however, said they were less likely to miss comments 
made during a discussion about the ideas and concepts taught in the course. 

• Students in partly online classes were more likely to discuss academic goals and course 
concepts with their instructor than students in fully online classes. 

• Students in fully online classes reported a greater likelihood of finding answers 
independently, than ask by asking the instructor or other students for help. 

• Students in fully online classes said they were more likely to put more thought into their 
comments as well as to ask awkward questions and spend more time studying than 
students in partly online classes. 

• Students in partly online classes said they had less difficulty relating to students in their 
class than students in fully online classes. 

• Students in fully online classes tended to give higher ratings to the technology than 
students in partly online classes. 

• Students in partly online classes were more likely to say that their instructor was excited 
about using the technology than students in fully online classes. 

• Both groups of students indicated they were either satisfied or very satisfied with their 
classes and would recommend the course and the instructor to other students. They also 
agreed that they would recommend that other students take classes with electronic 
communication. 

2. As may be expected, the students in the fully online classes rated their technical knowledge 
slightly higher than those in the partly online classes: 

• 86% of fully online (vs. 71% of the partly online) rated their ability as expert or near 
expert to send and receive email, 

• 75% of fully online (vs. 68%) rated their ability to search the Internet as expert or near 
expert, 

• 65% of the fully online students said they were expert or near expert in sending and 
receiving electronic files (vs. 45% of students in partly online classes). 
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SECTION III: Other Findings Comparing Specific Cohorts 

1. We examined how specific groups of students responded to the survey to see if there were 

any significant differences from the all-student survey results. We found a few differences: 

• Students enrolled in CIS classes indicated that they were more likely to apply what they 
learned to “real world” problems than the all student group. 

• CIS online students also had more confidence in the technologies they were using than 
the all student group. 

2. Gender differences, among students and faculty, may be significant: 

• Male students were more likely than female students to discuss ideas and concepts from 
the course with other students as well as work on assignments with other students. 

• Male students were more likely to ask other students for comments on their course work 
than female students. 

• Males appeared to be more willing to discuss ideas and concepts taught in the course with 
the instructor than females. 

• Males also said they had less difficulty relating to other students in the class and 
expressed greater levels of confidence that they could reach their academic goals. 

• Males rated themselves higher than females in their ability to search the Internet and to 
send and receive electronic files. 

3. Differences between male and female instructors also may be significant for some students: 

• Students were more likely to receive more detailed comments on their work from female 
instructors than male instructors (Q. 7, cross-tabs available). 

• Female instructors were also more likely to discuss academic goals with their students 
than male instructors (39% vs. 21% on Q. 10), and also more likely to discuss ideas and 
concepts with female rather than male instructors (48% vs. 34%, Q.l 1). 

• Students said they are more likely to apply real world problems when they have female 
instructors than with male instructors (53% vs. 44%, Q.13). 

Discussion of Findings 

Our findings are preliminary and must be replicated if they are to have substantial credibility. 
Preliminary indications are that students usually new to classes using electronic communication, 
say they are more likely to actively participate in classes with these tools than in face-to-face 
classes. They also indicate a greater willingness to ask clarifying questions when they don’t 
imderstand something, and they are more willing to discuss ideas and concepts with other 
students in the class — when using electronic commimication. These students say that email and 
online discussion has enhanced their relationship to the instructor. They are more likely to 
receive detailed comments from the instructor more promptly than in a face-to-face course. And 
they are more likely to tell the instructor that they have a complaint or suggestion than in a face- 
to-face course. Electronic communication, however, has its limits: students indicated they were 
less likely to discuss academic goals and career plans with their instructor and were less likely to 
discuss ideas and concepts taught in the course with their instructor. Students seem to be more 
autonomous, more willing search to search out their own answers to questions rather than rely 
upon the instructor. 
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The net result was that classes relying fully or partly on electronic communication generated a 
great deal of student enthusiasm. Significant majorities said they had put more thought into their 
comments, were willing to ask awkward questions, spent more time studying and three quarters 
said they were more confident that they would be able to reach their academic goals. Over ninety 
percent said they would recommend both the course and the instructor to other students. Ninety 
one percent said they would recommend that other students take classes with electronic 
communication. 

Comparing student responses based on whether they had enrolled in fully online or partly online 
classes suggests that the combination of face-to-face and electronic communication may be more 
useful and exciting for more students than the fully online classes. There is not enough data to 
make a strong case for this yet, but more study may yield some significant differences between 
the two modes of electronic delivery: fully online and combinations of online and class 

meetings. 

Discussion of Patterns in Open-Ended Comments 

The next two pages categorize the patterns in student responses to the open-ended questions, 
#32-34. The categorized comments compare students in fully online classes with those in partly 
online classes. Students often made more than one comment per question. The categories are 
entirely ours, based upon a thorough review of over 2,800 responses from 710 students. The 
categories with the highest number of responses suggest the following: 

Question 32: Imagine a course without electronic communication: how would it differ? 

• Students in fully online classes reported that without electronic communication, it would be 
less convenient or they would miss classes; they also might be unable to take the class, miss 
the information, Web research, and the discussions. 

• Students in partly online classes reported that without electronic communication they would 
miss the discussion, learn less and the course would be “a lot” different; it would be 
inconvenient and their attendance would be lower. 

Question 33: Greatest barriers to using various types of electronic communication 

• Students in fully online classes suggested that the primary barriers were unreliable servers, 
hardware/software problems, inexperience with computers, and slow instructor response 
time. It should be noted, however, that 70 students “could not think of any barriers” and 30 
students made such vague comments about barriers that they may have been unable to think 
of any. 

• Comments from those in partly online classes suggested that the primary barriers also 
included technical problems: access to computers, inexperience, and the inevitable “time.” 

Question 34: What else would you like us to know? 

• Students in fully online classes made positive comments on the whole; 209 comments were 
very positive. Some noted the technical and software problems that other survey questions 
cover. 
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• From the partly online classes, positive comments appeared to a lesser extent than from fully 
online students; however, many comments noted that these tools provided a convenient 
opportunity to get an education 

Fully Online Classes: N = 362 (of 710) 

NOTE: On some questions, students made more than one comment. 

Question 32: Imagine you do not have access to the electronic communications that were made available in this 
course, how would that change your educational experience? 



Comment Category 


Frequency 


Less convenient and/or I'd have lower attendance 


134 


Would not have been able to take this course and/or others 


69 


Miss key information, learn less, miss Web discussion, electronic communication 


67 


"Stuck with books" or have to use the library 


42 


No change or Don’t know 


18 


A lot different 


16 


Less student contact 


15 


Miscellaneous/Other/V ague responses 


12 


Less instructor contact 


7 


Not as much fun 


7 



Question 33: List below the three things that are the greatest barriers to your successful use of email, listservs and 
electronic bulletin boards used in this course (please list in order of priority). 



Comment Category 


Frequency 


Server down, slow, unreliable, course management software down 


75 


No barriers, none, no problems, can't think of any 


70 


Technical problems: software programs, difficulty, low reliability 


37 


Time: work, home, other courses limit involvement 


32 


Inexperience with computers, limited skills, typing, attachments 


30 


Miscellaneous/Other/V ague responses 


30 


Limitations of instructor: slow turn-around response 


21 


Low interest or dislike: anonymity, isolation, email itself, monitor weary 


20 


Limited home equipment or ISP (Internet Service Provider) too costly 


19 


Lack of self discipline or time management 


15 


No home computer (or have to use one at work) 


11 


Not applicable 


11 


Limited access to computers at school: labs full 


6 
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Question 34: What else would you like us to know about your experience with the technologies used in this 

course? 



Comment Category 


Frequency 


Enjoyed class, positive response 


91 


Convenient; provided greater opportunity to obtain an education 


59 


Great educational tool 


32 


Learned new computer technology skills 


27 


Nothing or Not applicable 


25 


Technical problems, glitches: server, TopClass, automatic quiz grader 


20 


Misc./OtherA^ague response 


14 


Too much to learn in too little time 


12 


Would like to see more courses offered online 


12 


Recommends online courses to others 


10 


Slow response from or not enough communication with instructor 


10 


Disliked online format, was confused 


8 


Not enough communication with other students 


4 


The technology was not used to its fullest potential 


2 



Partly Online Classes: N = 348 (of 710) 

NOTE: Students often made more than one comment per question. 

Question 32: Imagine you do not have access to the electronic communications that were made available in this 
course, how would that change your educational experience? 



Comment Category 


Frequency 


Miss key information, learn less; miss Web, discussion, electronic communication 


88 


Less convenient or I'd have lower attendance 


77 


No change or Don't know 


55 


Misc. and/or vague response 


30 


A lot different 


22 


"Stuck with books" or have to use library 


20 


Not as much fun 


16 


Would not have been able to take the course 


16 


Less student contact 


11 


Less instructor contact 


8 



Question 33: List below the three things that are the greatest barriers to your successjul use of email, listservs and 
electronic bulletin boards used in this course (please list in order of priority). 



Comment Category 


Frequency 


No barriers, none, no problems, can't think of any 


58 


Time: work, home, other courses limit involvement 


45 


Inexperience with computers, limited skills, typing, attachments, etc. 


39 


Misc, and/or vague response 


35 


No home computer (or have to use one at work) 


22 


Limited home equipment or ISP (Internet Service Provider) too costly 


19 


"Not applicable" 


17 


Limitations of instructor: slow turn-around responses, etc. 


16 


Server down, slow, unreliable, course management software down 


16 


Technical problems: software programs, difficulty, low reliability 


14 


Low interest or dislike: anonymity, isolation, email itself, monitor weary 


14 


Lack of self discipline or time management 


10 


Limited access to computers at school: labs full 


8 
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Question 34: What else would you like us to know about your experience with the technologies used in this 

course? 



Comment Category 


Frequency 


Convenient; provided greater opportunity to obtain an education 


45 


Nothing or Not applicable 


45 


Enioyed class, positive response 


44 


Miscellaneous and/or vague response 


41 


Great educational tool 


26 


Learned new computer technology skills 


23 


Too much to learn in too little time 


9 


Technical problems, glitches: server, TopClass, automatic quiz grader 


7 


Disliked online format, was confused 


6 


Would like to see more courses offered online 


5 


Recommends online courses to others 


5 


Not enough communication with other students 


5 


The technology was not used to its fullest potential 


4 


Slow response from or not enough communication with instructor 


4 



Implications 

We will need at least a second year of work to confirm these survey findings. To some extent, 

the project was a field test for finding out what we need to know. After thoroughly reviewing the 

data, the methodology, and the faculty reports, we have made some tentative conclusions about 

what subsequent project leaders should consider: 

1 . There is clear support by students for both online and web enhanced classes. 

2. The survey questions should be used again with perhaps some additions. It might include a 
comparison of student progress or success in face-to-face classes vs. online or web-enhanced 
classes (same curricula but different modes of delivery). 

3. The Project needs to capture the views of more students who drop out of the classes in the 
early part of the term; therefore, students should be surveyed both early and late in the term. 

4. The role of the college researchers was not made clear enough; some researchers on college 
teams were at a loss as how to help faculty thing through their assessment of learning little. 

5. More specific roles and responsibilities for the researchers would help them with one of our 
goals: “to engage faculty more in research and researchers more in teaching and learning.” 

6. We recommend strengthening the online exchange in any replication of the project. Faculty 
interest might have been higher if we had provided more provocative topics more 
consistently. 

7. Future projects might benefit from a finer distinction among classes as to amounts of web use 
and numbers of class meetings expected of students. While we separated “fully online” from 
all others in the data sets, drawing conclusions was a little tricky with this range. We 
suggest, however, that further studies should include this range, if only to make comparisons. 

8. All participating faculty must require students to fill out the Flashlight survey; uneven 
proportions of pilot class students limited validity of the results. 

9. The faculty recommended expanding workshops to two full days for both North and South in 
the Fall. Alternatively, one summit conference in the spring could discuss the data if the 
original faculty used the same survey items with a new group of students. 
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10. This project included a wide range of disciplines and programs; a few faculty suggested that 
we focus on a specific programs or disciplines next time. 

1 1 . Many faculty final reports and individual class assessments were submitted electronically; in 
the future we might require electronic reporting so that all reports and student responses 
(outside the survey) could be posted on the web site. (NOTE; To review an individual 
faculty report, request a copy through the faculty participant listed in the roster in Appendix 
H. Rio Hondo will provide a copy if permission is granted in writing. Contact Susan Obler 
at sobler@rh.cc.ca.us . 

12. While we asked participating faculty to choose a pilot course that had been taught at least 
once before, a few instructors were teaching in this delivery mode for the first time. In a 
future project, we may need to be more prescriptive about this expectation. In this project, 
however, the range of faculty experience was a benefit; it adds to the limitations of the survey 
and its results. 

13. Another project should require that the colleges have the technical capacity to support online 
or web-enhanced courses — at whatever degree of use — in order to reduce the fhistration 
some students and faculty experienced during this project. 
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